Vision-based scene detection

ABSTRACT

A method of distinguishing between daytime lighting conditions and nighttime lighting conditions based on a captured image by a vision-based imaging device along a path of travel. An image is captured by a vision-based imaging device. A region of interest is selected in the captured image. A light intensity value is determined for each pixel within the region of interest. A cumulative histogram is generated based on light intensity values within the region of interest. The cumulative histogram including a plurality of category bins representing the light intensity values. Each category bin identifies an aggregate value of light intensity values assigned to each respective category bin. An aggregate value is compared within a predetermined category bin of the histogram to a first predetermined threshold. A determination is made whether the image is captured during the daytime lighting conditions as a function of the aggregate value within the predetermined category bin.

BACKGROUND OF INVENTION

An embodiment relates generally to vehicle vision-based systems.

Vision-imaging systems are used in vehicles for enhancing sensingapplications within the vehicle such as vehicle detection systems, clearpath detection systems, lane centering, and other vision/positioningsystems. Such systems utilize a light sensor for distinguishing adaylight condition from a nighttime condition. A light sensor is anadded component that requires additional cost, wiring, and possiblecomplexity.

Many systems in a vehicle may utilize such light sensing information toswitch to a different mode of operation, or if the data obtained underthe respective lighting conditions is not robust, then the system mayswitch to other techniques for carrying out the mode of operation.Various in-vehicle systems have been used to detect the lightingconditions; however, methods such as classifiers that determine adaytime or nighttime condition are complex and data intensive.

SUMMARY OF INVENTION

An advantage of an embodiment is distinguishing a daytime environmentfrom a nighttime environment by analyzing images from an existingvehicle vision-based imaging device, which can eliminate a light sensor.The system can further distinguish a well-illuminated nighttimeenvironment from a poorly illuminated environment. The system furtheridentifies a daytime environment where a structure obstructs naturaldaylight in the captured image. This information can be used by variousvehicle applications for actuating or switching operating modes thatoperate based on the lighting condition exterior of the vehicle, or maybe inoperable when a signal blocking structure is present such as tunnelor under a bridge.

An embodiment contemplates a method of distinguishing between daytimelighting conditions and nighttime lighting conditions based on acaptured image by a vision-based imaging device along a path of travel.An image is captured by a vision-based imaging device. A region ofinterest is selected in the captured image. A light intensity value isdetermined for each pixel within the region of interest. A cumulativehistogram is generated based on light intensity values within the regionof interest. The cumulative histogram includes a plurality of categorybins representing the light intensity values. Each category binidentifies an aggregate value of light intensity values assigned to eachrespective category bin. An aggregate value is compared within apredetermined category bin of the histogram to a first predeterminedthreshold. A determination is made whether the image is captured duringthe daytime lighting conditions as a function of the aggregate valuewithin the predetermined category bin.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a plan view of a vehicle capturing an image of a road.

FIG. 2 is a view of an image by a captured by a vehicle vision-basedimaging device.

FIG. 3 is an example of an intensity histogram.

FIG. 4 is an example of a cumulative histogram.

FIG. 5 is an exemplary cumulative histogram of a daylight lightingenvironment.

FIG. 6 is an exemplary cumulative histogram of a nighttime lightingenvironment.

FIG. 7 is an exemplary cumulative histogram of a daytime lightingcondition that identifies a daylight blocking structure.

FIG. 8 is an exemplary cumulative histogram of a sufficientlyilluminated nighttime lighting environment.

FIG. 9 is a flowchart of a method distinguishing between variouslighting conditions.

DETAILED DESCRIPTION

There is shown in FIG. 1, a vehicle 10 traveling along a road 12. Avision-based imaging device 14 captures images exterior of the vehicle10 for detecting images along a path of travel. The vision-based imagingdevice may capture images in any region surrounding the vehicle, whichincludes, but is not limited to, images forward of the path of travel,images rearward of the path of travel, or images to a side of the pathof travel. The vision-based imaging device 14 is mounted on the vehicleso that the desired region along the path of travel is captured. In theembodiment shown in FIG. 1, the vision-based imaging device 14 ismounted just behind the front windshield for capturing events occurringexterior and forward of the vehicle. The vision-based imaging device 14is part of an existing system in the vehicle that is typically used forrecognition of road marking, lane markings, road signs, or other roadwayobjects used in lane departure warning systems and clear path detectionsystems. In the embodiments disclosed herein, the captured images fromthe vision-based imaging device 14 are used to distinguish between adaytime lighting condition and a nighttime lighting condition. Moreover,the captured images are also used to distinguish between an obstructeddaylight condition and an unobstructed daylight condition during thedaytime, and also between a well-illuminated environment and a poorlyilluminated environment during the nighttime. The identification of thedaylight condition is provided to vehicle applications which actuate orswitch operating modes based on the sensed lighting condition. As aresult, the determination of the lighting condition eliminates therequirement of a light sensing device while utilizing existing vehicleequipment.

FIG. 2 illustrates an image 16 captured by the vision-based imagingsystem on the vehicle. A region of interest 18 is selected within theimage 16 for analyzing the lighting condition. A skyline is preferablyselected as the region of interest 18 as the sky is the best indicatorfor determining the lighting condition. As a result, an upper portion ofthe image is selected for analyzing the lighting condition of the image.It should be understood that various software programs could be utilizedfor analyzing the image and selecting the region that would be bestsuited for detecting the lighting condition of the environment exteriorof the vehicle.

The in-vehicle technique for determining a lighting condition exteriorof the vehicle utilizes thresholds as opposed to a classifier which addscomplexity within a vehicle and is data intensive; however, thetechnique may utilize a classifier during the training states forestablishing the threshold. In the training mode, various images thatinclude various lighting conditions are captured. In each image, aregion of interest is identified that preferably relates to the skylineas shown in FIG. 2.

For each image captured, an intensity histogram is generated. FIG. 3illustrates an example of an intensity histogram. The region of interestof a respective image is analyzed. The image is produced from aplurality of pixels. Each pixel within the region of interest has anassociated light intensity value. Each of the light intensity values arerepresented within the histogram.

The histogram is segregated into a plurality of category bins. As shownin FIG. 3, there are sixteen category bins. Intensity values that areclose in intensity to one another may be grouped together. For example,if there are 256 possible light intensity values that a single pixel mayrepresent, then the histogram as illustrated in FIG. 3 is segregatedinto sixteen categories with sixteen light intensity values within eachcategory bin. Referring to FIG. 3, the first category bin will includelight intensity values 1-16, the second category bin will include 17-32,the third category bin will include 33-48, and so forth. As a result,each intensity value for each pixel within the region of interest isassigned to a category bin based on their respective category bin. Thevertical axis of the histogram represents an aggregate value of thepixels within the respective range of light intensity values assigned tothat category bin. The aggregate value may be an aggregate number, apercentage, or other representation that identifies the pixels assignedto that category bin.

After the intensity values for each pixel is assigned to a respectivecategory bin, a cumulative histogram is generated based on the intensityhistogram. An exemplary cumulative histogram is illustrated in FIG. 4. Arespective cumulative histogram is generated for each image foridentifying a respective category bin that best distinguishes a daytimelighting condition from a nighttime lighting condition. In addition, arespective category bin is identified that distinguishes obstructeddaylight conditions from unobstructed daylight conditions, andwell-illumination conditions from poor illumination conditions duringthe nighttime. Obstructed daylight conditions would include a pathtravel being driven under a structure that obstructs the daylight.Examples include, but are not limited to, tunnels and bridges.

As described earlier, a plurality of cumulative histograms are generatedthat represent both daytime lighting environments and nighttime lightingenvironments. The data from the cumulative histograms are provided to afeature selection routine for identifying the respective category binsthat best distinguish between a daytime lighting condition and anighttime lighting condition. An example of a feature selection routineincludes, but is not limited to, a Kullback-Leibler divergencemethodology that uses a non-symmetric measurement of the differencebetween two probability distributions. The objective of the routine isto collectively analyze each of the correlating category bins of thecumulative histograms and identify the respective category bin that bestdistinguishes the daytime lighting environment from the nighttimelighting environment. Once the respective category bin is identified, afirst threshold is selected for identifying whether the image isobtained from a daytime environment or a nighttime environment. Theaggregate value representing the light intensity value for the selectedcategory bin is compared to the first threshold. If the aggregate valueis less than the first threshold, then a determination is made thatimage is captured during a daytime lighting condition. Alternatively, ifthe aggregate value is greater than the first threshold, then thedetermination is made that the image is captured during a nighttimelighting condition.

FIGS. 5 and 6 illustrate respective cumulative histograms captured fromimages of both a daytime lighting environment and a nighttime lightingenvironment, respectively. A plurality of cumulative histograms may beused in the training mode for selecting the category bins and thethresholds. A feature selection program analyzes the cumulativehistograms collectively and determines the respective category bin thatbest distinguishes daytime lighting environment from the nighttimelighting environment based on the cumulative histograms. In the examplesshown, category bin 3 is selected and the first category bin by thefeature selection routine for distinguishing between the daytime andnighttime environment. In addition, a first threshold value of (e.g.,0.2) is selected as the comparative value of the first category bin iscompared to the aggregate value for determining whether the image iscaptured from the daytime lighting condition or the nighttime lightingcondition.

To provide further details of the lighting condition, a second categorybin is selected by the feature selection program that provides furtherinsight on whether the natural daylight during a daytime condition isbeing obstructed, or whether image is well-illuminated during anighttime lighting condition. The feature selection program analyzes thecumulative histograms of the training data and identifies the secondcategory bin that will be used to distinguish the above mentionedconditions.

For a daytime lighting condition, a second threshold is generated forcomparison with the aggregate value of the second category bin fordetermining whether the image includes a structure that obstructs thedaylight. If the aggregate value of the second category bin is greaterthan the second threshold, then the determination is made that thecaptured image includes a daylight blocking structure. If the aggregatevalue of the second category bin is less than the second threshold, thenthe determination is that the captured image has unobstructed daylightconditions. As shown in FIG. 5, category bin 5 is identified as thesecond category bin. The second category bin is less than the secondthreshold. Therefore, the determination is made that there is nostructure obstructing the daylight.

Alternatively, FIG. 7 illustrates a daytime lighting condition thatincludes a structure obstructing the natural daylight. As shown in thehistogram of FIG. 7, the aggregate value of the second category bin isless than the second threshold (e.g., 0.7), which indicates that adaytime lighting condition is present. However, the aggregate value ofthe second category bin is greater than the second threshold (e.g.,0.7). Therefore, the determination is made that the daylight blockingstructures are present in the image.

For a nighttime lighting condition, the second category bin is used todetermine whether the captured image is sufficiently illuminated duringthe nighttime condition. A third threshold is generated for comparisonwith the aggregate value of the second category bin for determiningwhether the image is sufficiently illuminated. If the aggregate value inthe second category bin is less than the third threshold, then adetermination is made that the image is sufficiently illuminated. If theaggregate value in the second category bin is greater than the thirdthreshold, then a determination is made that the image is notwell-illuminated. Referring FIG. 6, the fifth category bin is greaterthan the third threshold. Therefore, the determination is made thatscene within the capture image is not well-illuminated.

FIG. 8 illustrates nighttime lighting condition that is sufficientlyilluminated by artificial lighting. Referring to the histogram, thethird category bin is greater than the second threshold (e.g., 0.2).Therefore, the image is identified as a nighttime condition. The fifthcategory bin indicates that the aggregate value is less than thirdthreshold (e.g. 0.7). Therefore, the determination is made that thescene in the captured image is sufficiently illuminated.

After the training mode is complete and the category bins and thresholdsare identified, the routine may be implemented in the vehicle or anyother apparatus that utilizes light sensing conditions. The advantage ofthe technique described herein, is that category bins and thresholdsthat are derived in a training mode are utilized in a vehicle real-timesetting for identifying the daytime or nighttime mode as opposed toutilizing classifiers which are complex and often data intensive. Byutilizing thresholds, the analysis can be performed in real-time and theresults can be provided immediately to other vehicle applications whichrely on light sensing analysis.

FIG. 9 illustrates a flow diagram of the vision scene detection processapplied in a vehicle. In block 20, a captured image is obtained from avision-based imaging device mounted on the vehicle. The image depicts aview of the path of travel of the vehicle; however, the image isutilized for determining the daylight condition of the environmentexterior of the vehicle.

In block 21, a region of interest is selected in the image. The regionof interest is preferably a region that includes the skyline, whichwould provide the most information relating to the light conditions ofthe exterior environment.

In block 22, each pixel that comprises the region of interest isanalyzed by identifying the light intensity value of each pixel. Basedon a light intensity value, each pixel is assigned to a respectivecategory bin that correlates to the light intensity value for generatinga cumulative histogram. It should be understood that the all of thecategory bins are not required to be constructed in the cumulativehistogram. Rather, only the respective category bins that include up tothe second predetermined category bin are required. The reason for thisis that remaining category bins beyond the second predetermined bin arenot utilized. Therefore, the time to construct the histogram is reducedsince none of the category bins beyond the second category bin isutilized. After the cumulative histogram is constructed, the firstpredetermined category bin (e.g., 3) and the second category bin (e.g.,5) are identified in the histogram.

In block 23, a determination is made as to whether the aggregate valuerepresenting the first category bin is greater than a firstpredetermined threshold. If the determination is made that firstaggregate value is less than a first threshold, then a determination ismade that a daytime lighting environment is present and the routineproceeds to block 24. If the determination is made that the firstaggregate value is greater than the first threshold, then thedetermination is made that a nighttime lighting environment is presentand the routine proceeds to block 27.

In block 24, a determination is made whether the second aggregate valueis greater than a second threshold. If the second aggregate value isgreater than the second threshold, then the routine proceeds to block25. In block 25, the determination is made that the captured imageincludes a structure that obstructs daylight, and the vehicle outputsinformation regarding the obstructed daylight condition to other vehicleapplications that can utilize this information accordingly. Thepertinence of determining a structure that obstructs daylight is toassist a vehicle application in determining at what point in the road oftravel is the obstruction expected. For example, Global PositioningSystem (GPS) or other Global Navigation Satellite System (GNSS)receivers operate by tracking line of sight signals. These receiverstypically require at least four or more satellites to be continuouslyavailable in an unobstructed line of sight of a satellite receiver on avehicle. Due to natural and man-made obstructions (e.g., bridge ortunnels), the theoretical minimum number of satellites required toaccurately determine a position of the satellite receiver may not beavailable under certain conditions. When a vehicle GPS receiver loosescommunication with the respective satellites due to obstruction fromsuch structures, other data and techniques may be used to compensate forlocation error increase as a result of poor GPS accuracy. As a result,it would be helpful to know approximately when the GPS signal will belost due to the obstruction. What would be beneficial if the GPSreceiver could be forewarned of the loss of signal, so that it could usealternative means for tracking the vehicle just before reaching thestructure and while driving under the structure.

In another example, systems that utilize the light intensity of theexterior environment information (e.g., clear path detection) to adjusttheir operating mode to compensate for lack of illumination. Suchsystems can eliminate a light sensing device and utilize the informationprovided by the vision scene detection system.

Referring again to block 24, if the second aggregate value is less thanthe second threshold, then the routine proceeds to block 26. In block26, no action is taken as the vehicle is traveling in a daytime lightenvironment with no obstructions.

In block 27, in response to the determination being made that anighttime lighting environment is present, a determination is madewhether the second aggregate value is less than a third threshold. Ifthe second aggregate value is less than the third threshold, then theroutine proceeds to step 28. If the second aggregate threshold isgreater than the third threshold, then the routine proceeds to step 29.

In step 28, in response to the second aggregate value being less thanthe third threshold, a determination is made that the nighttime lightingenvironment is sufficiently illuminated. The vision-based scenedetection system outputs the lighting condition to the other vehicleapplications to utilize accordingly.

In step 29, in response to the second aggregate value being greater thanthe third threshold, a determination is made that the nighttime lightingenvironment is includes insufficient lighting conditions. As a result,the vision-based scene detection system outputs the lighting conditionto the other vehicle applications to utilize accordingly.

While certain embodiments of the present invention have been describedin detail, those familiar with the art to which this invention relateswill recognize various alternative designs and embodiments forpracticing the invention as defined by the following claims.

What is claimed is:
 1. A method of distinguishing between daytimelighting conditions and nighttime lighting conditions based on acaptured image by a vision-based imaging device along a path of travel,the method comprising the steps of: capturing an image by a vision-basedimaging device; selecting a region of interest in the captured image;determining a light intensity value for each pixel within the region ofinterest; generating a cumulative histogram based on light intensityvalues within the region of interest, the cumulative histogram includinga plurality of category bins representing the light intensity values,each category bin identifying an aggregate value of light intensityvalues assigned to each respective category bin; comparing an aggregatevalue within a predetermined category bin of the histogram to a firstpredetermined threshold; and determining whether the image is capturedduring the daytime lighting conditions as a function of the aggregatevalue within the predetermined category bin.
 2. The method of claim 1wherein the determination that the image is captured during the daylightlighting conditions is based on the aggregate number being less than afirst predetermined threshold.
 3. The method of claim 2 wherein thedetermination of whether the captured image is obtained during thedaytime lighting conditions further comprises the step of determiningwhether the captured image includes a daylight blocking structure overthe path of travel.
 4. The method of claim 3 wherein determining whetherthe captured image includes a daylight blocking structure over the pathof travel comprises: comparing an aggregate value of the light intensityvalues within the second predetermined category bin of the cumulativehistogram to a second predetermined threshold; determining whether theaggregate value of the light intensity values within the secondpredetermined category bin is greater than the second predeterminedthreshold; and determining that the captured image includes the daylightblocking structure over the path of travel in response to the aggregatevalue within the second predetermined category bin being greater thanthe predetermined threshold.
 5. The method of claim 4 wherein thedetermination that the captured image includes the daylight blockingstructure over the path of travel is output to a global positioningsystem, wherein the global positioning system estimates a time when aloss of signal occurs in the global positioning system as a result ofvehicle traveling under the daylight blocking structure.
 6. The methodof claim 3 wherein the daylight blocking structure over the path oftravel includes a tunnel over the path of travel.
 7. The method of claim3 wherein the daylight blocking structure over the path of travelincludes a bridge over the path of travel.
 8. The method of claim 1wherein a determination that the captured image is obtained during thenighttime lighting conditions is based on the aggregate number beinggreater than the first predetermined threshold.
 9. The method of claim 8wherein determining that the capture image is obtained during nighttimelighting conditions further comprises the step of determining whetherthe nighttime lighting conditions include substantial lighting forillumination the path of travel in response to the aggregate value beinggreater than the first predetermined threshold, wherein determining anillumination of the path of travel comprises: comparing an aggregatevalue of a second predetermined category bin of the cumulative histogramto a third predetermined threshold; determining whether an aggregatevalue of the light intensity values within the second predeterminedcategory bin is less than the third predetermined threshold; anddetermining that the captured image includes the path of travel in thecaptured image is substantially illuminated in response to the secondpredetermined category being less than the third predeterminedthreshold.
 10. The method of claim 1 wherein the determined daytime andnighttime lighting conditions is output to a vehicle-based module thatenables vehicle operations based on environment light sensing.
 11. Themethod of claim 10 wherein the vehicle-based module includes autoheadlamps wherein the output is used to enable auto headlamps.
 12. Themethod of claim 10 wherein the vehicle-based module includes a vehiclevision system, and wherein output is used to modify the vehicle visionsystem.
 13. The method of claim 1 wherein the selected region ofinterest represents an expected location of a skyline within thecaptured image.
 14. The method of claim 1 wherein capturing an image bya vision-based imaging device includes capturing multiple images over aperiod of time, wherein the multiple images are cumulatively analyzedfor detecting the daytime lighting conditions.
 15. The method of claim14 wherein the multiple images are used to determine a transitioningregion in a road of travel between an unobstructed daylight portion ofthe road of travel and an obstructed daylight portion of the road oftravel.
 16. The method of claim 1 wherein a training phase is used toidentify which of the respective category bins of the cumulativehistogram are selected as the predetermined category bin and a secondpredetermined category bin, wherein the training phase comprises:capturing a plurality of images of various lighting conditions;selecting a region of interest for each captured image; determining alight intensity value for each pixel within the region of interest foreach captured image; generating a cumulative histogram for each imagebased on the light intensity values within the region of interest, eachcumulative histogram including a plurality category bins representingthe light intensity values, each category bin identified by acomparative factor that is derived as function of the light intensityvalues assigned to each respective category bin; identifying a firstcategory bin from the plurality of histograms that distinguishes adaytime lighting condition from a nighttime lighting condition based oncomparative factors, the first category bin being selected as thepredetermined category bin; identifying a second category bin from theplurality of histograms that distinguishes a daytime lighting conditionhaving a daylight blocking structure from an unobstructed daytimelighting condition based on the comparative factors, the second categorybin being selected as the second predetermined category bin.
 17. Themethod of claim 16 wherein the predetermined category bin is determinedby identifying a largest difference between a comparative factor of arespective category bin of a histogram associated with a daytimelighting condition and a comparative factor of a correlating categorybin of a histogram associated with the nighttime lighting condition. 18.The method of claim 17 wherein the second predetermined category bin isdetermined by identifying a largest difference between a comparativefactor of a respective category bin of a histogram associated with anunobstructed daylight condition and a comparative factor of acorrelating category bin of a histogram associated with the unobstructeddaylight condition.
 19. The method of claim 16 wherein the predeterminedcategory bin and the second predetermined category bin are selectedusing a feature selection technique.
 20. The method of claim 16 whereinthe comparative value is an aggregate value of the light intensityvalues within a respective category bin.